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Status of This Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
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Abstract

   The purpose of this report is to document how the requirements for
   publication of a routing protocol as an Internet Draft Standard have
   been satisfied by Border Gateway Protocol version 4 (BGP-4).

   This report satisfies the requirement for "the second report", as
   described in Section 6.0 of RFC 1264.  In order to fulfill the
   requirement, this report augments RFC 1774 and summarizes the key
   features of BGP-4, as well as analyzes the protocol with respect to
   scaling and performance.
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1.  Introduction

   BGP-4 is an inter-autonomous system routing protocol designed for
   TCP/IP internets.  Version 1 of BGP-4 was published in [RFC1105].
   Since then, BGP versions 2, 3, and 4 have been developed.  Version 2
   was documented in [RFC1163].  Version 3 is documented in [RFC1267].
   Version 4 is documented in [BGP4] (version 4 of BGP will hereafter be
   referred to as BGP).  The changes between versions are explained in
   Appendix A of [BGP4].  Possible applications of BGP in the Internet
   are documented in [RFC1772].

   BGP introduced support for Classless Inter-Domain Routing (CIDR)
   [RFC1519].  Because earlier versions of BGP lacked the support for
   CIDR, they are considered obsolete and unusable in today's Internet.

   The purpose of this report is to document how the requirements for
   publication of a routing protocol as an Internet Draft Standard have
   been satisfied by Border Gateway Protocol version 4 (BGP-4).

   This report satisfies the requirement for "the second report", as
   described in Section 6.0 of [RFC1264].  In order to fulfill the
   requirement, this report augments [RFC1774] and summarizes the key
   features of BGP-4, as well as analyzes the protocol with respect to
   scaling and performance.
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2.  Key Features and Algorithms of BGP

   This section summarizes the key features and algorithms of BGP.  BGP
   is an inter-autonomous system routing protocol; it is designed to be
   used between multiple autonomous systems.  BGP assumes that routing
   within an autonomous system is done by an intra-autonomous system
   routing protocol.  BGP also assumes that data packets are routed from
   source towards destination independent of the source.  BGP does not
   make any assumptions about intra-autonomous system routing protocols
   deployed within the various autonomous systems.  Specifically, BGP
   does not require all autonomous systems to run the same intra-
   autonomous system routing protocol (i.e., interior gateway protocol
   or IGP).

   Finally, note that BGP is a real inter-autonomous system routing
   protocol; and, as such, it imposes no constraints on the underlying
   interconnect topology of the autonomous systems.  The information
   exchanged via BGP is sufficient to construct a graph of autonomous
   systems connectivity from which routing loops may be pruned, and many
   routing policy decisions at the autonomous system level may be
   enforced.

2.1.  Key Features

   The key features of the protocol are the notion of path attributes
   and aggregation of Network Layer Reachability Information (NLRI).

   Path attributes provide BGP with flexibility and extensibility.  Path
   attributes are either well-known or optional.  The provision for
   optional attributes allows experimentation that may involve a group
   of BGP routers without affecting the rest of the Internet.  New
   optional attributes can be added to the protocol in much the same way
   that new options are added to, for example, the Telnet protocol
   [RFC854].

   One of the most important path attributes is the Autonomous System
   Path, or AS_PATH.  As the reachability information traverses the
   Internet, this (AS_PATH) information is augmented by the list of
   autonomous systems that have been traversed thus far, forming the
   AS_PATH.  The AS_PATH allows straightforward suppression of the
   looping of routing information.  In addition, the AS_PATH serves as a
   powerful and versatile mechanism for policy-based routing.

   BGP enhances the AS_PATH attribute to include sets of autonomous
   systems as well as lists via the AS_SET attribute.  This extended
   format allows generated aggregate routes to carry path information
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   from the more specific routes used to generate the aggregate.  It
   should be noted, however, that as of this writing, AS_SETs are rarely
   used in the Internet [ROUTEVIEWS].

2.2.  BGP Algorithms

   BGP uses an algorithm that is neither a pure distance vector
   algorithm or a pure link state algorithm.  Instead, it uses a
   modified distance vector algorithm, referred to as a "Path Vector"
   algorithm.  This algorithm uses path information to avoid traditional
   distance vector problems.  Each route within BGP pairs information
   about the destination with path information to that destination.
   Path information (also known as AS_PATH information) is stored within
   the AS_PATH attribute in BGP.  The path information assists BGP in
   detecting AS loops, thereby allowing BGP speakers to select loop-free
   routes.

   BGP uses an incremental update strategy to conserve bandwidth and
   processing power.  That is, after initial exchange of complete
   routing information, a pair of BGP routers exchanges only the changes
   to that information.  Such an incremental update design requires
   reliable transport between a pair of BGP routers in order to function
   correctly.  BGP solves this problem by using TCP for reliable
   transport.

   In addition to incremental updates, BGP has added the concept of
   route aggregation so that information about groups of destinations
   that use hierarchical address assignment (e.g., CIDR) may be
   aggregated and sent as a single Network Layer Reachability (NLRI).

   Finally, note that BGP is a self-contained protocol.  That is, BGP
   specifies how routing information is exchanged, both between BGP
   speakers in different autonomous systems, and between BGP speakers
   within a single autonomous system.

2.3.  BGP Finite State Machine (FSM)

   The BGP FSM is a set of rules that is applied to a BGP speaker's set
   of configured peers for the BGP operation.  A BGP implementation
   requires that a BGP speaker must connect to and listen on TCP port
   179 for accepting any new BGP connections from its peers.  The BGP
   Finite State Machine, or FSM, must be initiated and maintained for
   each new incoming and outgoing peer connection.  However, in steady
   state operation, there will be only one BGP FSM per connection per
   peer.
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   There may be a short period during which a BGP peer may have separate
   incoming and outgoing connections resulting in the creation of two
   different BGP FSMs relating to a peer (instead of one).  This can be
   resolved by following the BGP connection collision rules defined in
   the [BGP4] specification.

   The BGP FSM has the following states associated with each of its
   peers:

   IDLE:           State when BGP peer refuses any incoming connections.

   CONNECT:        State in which BGP peer is waiting for its TCP
                   connection to be completed.

   ACTIVE:         State in which BGP peer is trying to acquire a peer
                   by listening and accepting TCP connection.

   OPENSENT:       BGP peer is waiting for OPEN message from its peer.

   OPENCONFIRM:    BGP peer is waiting for KEEPALIVE or NOTIFICATION
                   message from its peer.

   ESTABLISHED:    BGP peer connection is established and exchanges
                   UPDATE, NOTIFICATION, and KEEPALIVE messages with its
                   peer.

   There are a number of BGP events that operate on the above mentioned
   states of the BGP FSM for BGP peers.  Support of these BGP events is
   either mandatory or optional.  These events are triggered by the
   protocol logic as part of the BGP or by using an operator
   intervention via a configuration interface to the BGP protocol.

   These BGP events are of following types: Optional events linked to
   Optional Session attributes, Administrative Events, Timer Events, TCP
   Connection-based Events, and BGP Message-based Events.  Both the FSM
   and the BGP events are explained in detail in [BGP4].

3.  BGP Capabilities

   The BGP capability mechanism [RFC3392] provides an easy and flexible
   way to introduce new features within the protocol.  In particular,
   the BGP capability mechanism allows a BGP speaker to advertise to its
   peers during startup various optional features supported by the
   speaker (and receive similar information from the peers).  This
   allows the base BGP to contain only essential functionality, while
   providing a flexible mechanism for signaling protocol extensions.
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4.  BGP Persistent Peer Oscillations

   Whenever a BGP speaker detects an error in a peer connection, it
   shuts down the peer and changes its FSM state to IDLE.  BGP speaker
   requires a Start event to re-initiate an idle peer connection.  If
   the error remains persistent and BGP speaker generates a Start event
   automatically, then it may result in persistent peer flapping.
   Although peer oscillation is found to be wide-spread in BGP
   implementations, methods for preventing persistent peer oscillations
   are outside the scope of base BGP specification.

5.  Implementation Guidelines

   A robust BGP implementation is "work conserving".  This means that if
   the number of prefixes is bounded, arbitrarily high levels of route
   change can be tolerated.  High levels can be tolerated with bounded
   impact on route convergence for occasional changes in generally
   stable routes.

   A robust implementation of BGP should have the following
   characteristics:

      1.  It is able to operate in almost arbitrarily high levels of
          route flap without losing peerings (failing to send
          keepalives) or losing other protocol adjacencies as a result
          of BGP load.

      2.  Instability of a subset of routes should not affect the route
          advertisements or forwarding associated with the set of stable
          routes.

      3.  Instability should not be caused by peers with high levels of
          instability or with different CPU speed or load that result in
          faster or slower processing of routes.  These instable peers
          should have a bounded impact on the convergence time for
          generally stable routes.

   Numerous robust BGP implementations exist.  Producing a robust
   implementation is not a trivial matter, but is clearly achievable.

6.  BGP Performance Characteristics and Scalability

   In this section, we provide "order of magnitude" answers to the
   questions of how much link bandwidth, router memory and router CPU
   cycles BGP will consume under normal conditions.  In particular, we
   will address the scalability of BGP and its limitations.
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6.1.  Link Bandwidth and CPU Utilization

   Immediately after the initial BGP connection setup, BGP peers
   exchange complete sets of routing information.  If we denote the
   total number of routes in the Internet as N, the total path
   attributes (for all N routes) received from a peer as A, and assume
   that the networks are uniformly distributed among the autonomous
   systems, then the worst-case amount of bandwidth consumed during the
   initial exchange between a pair of BGP speakers (P) is

           BW = O((N + A) * P)

   BGP-4 was created specifically to reduce the size of the set of NLRI
   entries, which has to be carried and exchanged by border routers.
   The aggregation scheme, defined in [RFC1519], describes the
   provider-based aggregation scheme in use in today's Internet.

   Due to the advantages of advertising a few large aggregate blocks
   (instead of many smaller class-based individual networks), it is
   difficult to estimate the actual reduction in bandwidth and
   processing that BGP-4 has provided over BGP-3.  If we simply
   enumerate all aggregate blocks into their individual, class-based
   networks, we would not take into account "dead" space that has been
   reserved for future expansion.  The best metric for determining the
   success of BGP's aggregation is to sample the number NLRI entries in
   the globally-connected Internet today, and compare it to growth rates
   that were projected before BGP was deployed.

   At the time of this writing, the full set of exterior routes carried
   by BGP is approximately 134,000 network entries [ROUTEVIEWS].

6.1.1.  CPU Utilization

   An important and fundamental feature of BGP is that BGP's CPU
   utilization depends only on the stability of its network which
   relates to BGP in terms of BGP UPDATE message announcements.  If the
   BGP network is stable, all the BGP routers within its network are in
   the steady state.  Thus, the only link bandwidth and router CPU
   cycles consumed by BGP are due to the exchange of the BGP KEEPALIVE
   messages.  The KEEPALIVE messages are exchanged only between peers.
   The suggested frequency of the exchange is 30 seconds.  The KEEPALIVE
   messages are quite short (19 octets) and require virtually no
   processing.  As a result, the bandwidth consumed by the KEEPALIVE
   messages is about 5 bits/sec.  Operational experience confirms that
   the overhead (in terms of bandwidth and CPU) associated with the
   KEEPALIVE messages should be viewed as negligible.
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   During periods of network instability, BGP routers within the network
   are generating routing updates that are exchanged using the BGP
   UPDATE messages.  The greatest overhead per UPDATE message occurs
   when each UPDATE message contains only a single network.  It should
   be pointed out that, in practice, routing changes exhibit strong
   locality with respect to the route attributes.  That is, routes that
   change are likely to have common route attributes.  In this case,
   multiple networks can be grouped into a single UPDATE message, thus
   significantly reducing the amount of bandwidth required (see also
   Appendix F.1 of [BGP4]).

6.1.2.  Memory Requirements

   To quantify the worst-case memory requirements for BGP, we denote the
   total number of networks in the Internet as N, the mean AS distance
   of the Internet as M (distance at the level of an autonomous system,
   expressed in terms of the number of autonomous systems), the total
   number of unique AS paths as A.  Then the worst-case memory
   requirements (MR) can be expressed as

           MR = O(N + (M * A))

   Because a mean AS distance M is a slow moving function of the
   interconnectivity ("meshiness") of the Internet, for all practical
   purposes the worst-case router memory requirements are on the order
   of the total number of networks in the Internet multiplied by the
   number of peers that the local system is peering with.  We expect
   that the total number of networks in the Internet will grow much
   faster than the average number of peers per router.  As a result,
   BGP's memory-scaling properties are linearly related to the total
   number of networks in the Internet.

   The following table illustrates typical memory requirements of a
   router running BGP.  We denote the average number of routes
   advertised by each peer as N, the total number of unique AS paths as
   A, the mean AS distance of the Internet as M (distance at the level
   of an autonomous system, expressed in terms of the number of
   autonomous systems), the number of octets required to store a network
   as R, and the number of bytes required to store one AS in an AS path
   as P.  It is assumed that each network is encoded as four bytes, each
   AS is encoded as two bytes, and each networks is reachable via some
   fraction of all the peers (# BGP peers/per net).  For purposes of the
   estimates here, we will calculate MR = (((N * R) + (M * A) * P) * S).
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   # Networks  Mean AS Distance # ASes # BGP peers/per net   Memory Req
       (N)             (M)        (A)          (P)              (MR)
   ----------  ---------------- ------ ------------------- -------------
     100,000           20         3,000         20           10,400,000
     100,000           20        15,000         20           20,000,000
     120,000           10        15,000        100           78,000,000
     140,000           15        20,000        100          116,000,000

   In analyzing BGP's memory requirements, we focus on the size of the
   BGP RIB table (ignoring implementation details).  In particular, we
   derive upper bounds for the size of the BGP RIB table.  For example,
   at the time of this writing, the BGP RIB tables of a typical backbone
   router carry on the order of 120,000 entries.  Given this number, one
   might ask whether it would be possible to have a functional router
   with a table containing 1,000,000 entries.  Clearly, the answer to
   this question is more related to how BGP is implemented.  A robust
   BGP implementation with a reasonable CPU and memory should not have
   issues scaling to such limits.

7.  BGP Policy Expressiveness and its Implications

   BGP is unique among deployed IP routing protocols in that routing is
   determined using semantically rich routing policies.  Although
   routing policies are usually the first BGP issue that comes to a
   network operator's mind, it is important to note that the languages
   and techniques for specifying BGP routing policies are not part of
   the protocol specification (see [RFC2622] for an example of such a
   policy language).  In addition, the BGP specification contains few
   restrictions, explicit or implicit, on routing policy languages.
   These languages have typically been developed by vendors and have
   evolved through interactions with network engineers in an environment
   lacking vendor-independent standards.

   The complexity of typical BGP configurations, at least in provider
   networks, has been steadily increasing.  Router vendors typically
   provide hundreds of special commands for use in the configuration of
   BGP, and this command set is continually expanding.  For example, BGP
   communities [RFC1997] allow policy writers to selectively attach tags
   to routes and to use these to signal policy information to other
   BGP-speaking routers.  Many providers allow customers, and sometimes
   peers, to send communities that determine the scope and preference of
   their routes.  Due to these developments, the task of writing BGP
   configurations has increasingly more aspects associated with open-
   ended programming.  This has allowed network operators to encode
   complex policies in order to address many unforeseen situations, and
   has opened the door for a great deal of creativity and
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   experimentation in routing policies.  This policy flexibility is one
   of the main reasons why BGP is so well suited to the commercial
   environment of the current Internet.

   However, this rich policy expressiveness has come with a cost that is
   often not recognized.  In particular, it is possible to construct
   locally defined routing policies that can lead to protocol divergence
   and unexpected global routing anomalies such as (unintended) non-
   determinism.  If the interacting policies causing such anomalies are
   defined in different autonomous systems, then these problems can be
   very difficult to debug and correct.  In the following sections, we
   describe two such cases relating to the existence (or lack thereof)
   of stable routings.

7.1.  Existence of Unique Stable Routings

   One can easily construct sets of policies for which BGP cannot
   guarantee that stable routings are unique.  This is illustrated by
   the following simple example.  Consider four Autonomous Systems, AS1,
   AS2, AS3, and AS4.  AS1 and AS2 are peers, and they provide transit
   for AS3 and AS4, respectively.  Suppose AS3 provides transit for AS4
   (in this case AS3 is a customer of AS1, and AS4 is a multihomed
   customer of both AS3 and AS2).  AS4 may want to use the link to AS3
   as a "backup" link, and sends AS3 a community value that AS3 has
   configured to lower the preference of AS4's routes to a level below
   that of its upstream provider, AS1.  The intended "backup routing" to
   AS4 is illustrated here:

              AS1 ------> AS2
              /|\          |
               |           |
               |           |
               |          \|/
              AS3 ------- AS4

   That is, the AS3-AS4 link is intended to be used only when the AS2-
   AS4 link is down (for outbound traffic, AS4 simply gives routes from
   AS2 a higher local preference).  This is a common scenario in today's
   Internet.  But note that this configuration has another stable
   solution:

              AS1 ------- AS2
               |           |
               |           |
               |           |
              \|/         \|/
              AS3 ------> AS4
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   In this case, AS3 does not translate the "depref my route" community
   received from AS4 into a "depref my route" community for AS1.
   Therefore, if AS1 hears the route to AS4 that transits AS3, it will
   prefer that route (because AS3 is a customer).  This state could be
   reached, for example, by starting in the "correct" backup routing
   shown first, bringing down the AS2-AS4 BGP session, and then bringing
   it back up.  In general, BGP has no way to prefer the "intended"
   solution over the anomalous one.  The solution picked will depend on
   the unpredictable order of BGP messages.

   While this example is relatively simple, many operators may fail to
   recognize that the true source of the problem is that the BGP
   policies of ASes can interact in unexpected ways, and that these
   interactions can result in multiple stable routings.  One can imagine
   that the interactions could be much more complex in the real
   Internet.  We suspect that such anomalies will only become more
   common as BGP continues to evolve with richer policy expressiveness.
   For example, extended communities provide an even more flexible means
   of signaling information within and between autonomous systems than
   is possible with [RFC1997] communities.  At the same time,
   applications of communities by network operators are evolving to
   address complex issues of inter-domain traffic engineering.

7.2.  Existence of Stable Routings

   One can also construct a set of policies for which BGP cannot
   guarantee that a stable routing exists (or, worse, that a stable
   routing will ever be found).  For example, [RFC3345] documents
   several scenarios that lead to route oscillations associated with the
   use of the Multi-Exit Discriminator (MED) attribute.  Route
   oscillation will happen in BGP when a set of policies has no
   solution.  That is, when there is no stable routing that satisfies
   the constraints imposed by policy, BGP has no choice but to keep
   trying.  In addition, even if BGP configurations can have a stable
   routing, the protocol may not be able to find it; BGP can "get
   trapped" down a blind alley that has no solution.

   Protocol divergence is not, however, a problem associated solely with
   use of the MED attribute.  This potential exists in BGP even without
   the use of the MED attribute.  Hence, like the unintended
   nondeterminism described in the previous section, this type of
   protocol divergence is an unintended consequence of the unconstrained
   nature of BGP policy languages.
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8.  Applicability

   In this section we identify the environments for which BGP is well
   suited, and the environments for which it is not suitable.  This
   question is partially answered in Section 2 of BGP [BGP4], which
   states:

      "To characterize the set of policy decisions that can be enforced
      using BGP, one must focus on the rule that an AS advertises to its
      neighbor ASes only those routes that it itself uses.  This rule
      reflects the "hop-by-hop" routing paradigm generally used
      throughout the current Internet.  Note that some policies cannot
      be supported by the "hop-by-hop" routing paradigm and thus require
      techniques such as source routing to enforce.  For example, BGP
      does not enable one AS to send traffic to a neighbor AS intending
      that the traffic take a different route from that taken by traffic
      originating in the neighbor AS.  On the other hand, BGP can
      support any policy conforming to the "hop-by-hop" routing
      paradigm.  Since the current Internet uses only the "hop-by-hop"
      routing paradigm and since BGP can support any policy that
      conforms to that paradigm, BGP is highly applicable as an inter-AS
      routing protocol for the current Internet."

   One of the important points here is that BGP contains only essential
   functionality, while at the same time providing a flexible mechanism
   within the protocol that allows us to extend its functionality.  For
   example, BGP capabilities provide an easy and flexible way to
   introduce new features within the protocol.  Finally, because BGP was
   designed to be flexible and extensible, new and/or evolving
   requirements can be addressed via existing mechanisms.

   To summarize, BGP is well suited as an inter-autonomous system
   routing protocol for any internet that is based on IP [RFC791] as the
   internet protocol and the "hop-by-hop" routing paradigm.
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10.  Security Considerations

   BGP provides flexible mechanisms with varying levels of complexity
   for security purposes.  BGP sessions are authenticated using BGP
   session addresses and the assigned AS number.  Because BGP sessions
   use TCP (and IP) for reliable transport, BGP sessions are further
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   authenticated and secured by any authentication and security
   mechanisms used by TCP and IP.

   BGP uses TCP MD5 option for validating data and protecting against
   spoofing of TCP segments exchanged between its sessions.  The usage
   of TCP MD5 option for BGP is described at length in [RFC2385].  The
   TCP MD5 Key management is discussed in [RFC3562].  BGP data
   encryption is provided using the IPsec mechanism, which encrypts the
   IP payload data (including TCP and BGP data).  The IPsec mechanism
   can be used in both the transport mode and the tunnel mode.  The
   IPsec mechanism is described in [RFC2406].  Both the TCP MD5 option
   and the IPsec mechanism are not widely deployed security mechanisms
   for BGP in today's Internet.  Hence, it is difficult to gauge their
   real performance impact when using with BGP.  However, because both
   the mechanisms are TCP- and IP-based security mechanisms, the Link
   Bandwidth, CPU utilization and router memory consumed by BGP would be
   the same as any other TCP- and IP-based protocols.

   BGP uses the IP TTL value to protect its External BGP (EBGP) sessions
   from any TCP- or IP-based CPU-intensive attacks.  It is a simple
   mechanism that suggests the use of filtering BGP (TCP) segments,
   using the IP TTL value carried within the IP header of BGP (TCP)
   segments that are exchanged between the EBGP sessions.  The BGP TTL
   mechanism is described in [RFC3682].  Usage of [RFC3682] impacts
   performance in a similar way as using any access control list (ACL)
   policies for BGP.

   Such flexible TCP- and IP-based security mechanisms, allow BGP to
   prevent insertion/deletion/modification of BGP data, any snooping of
   the data, session stealing, etc.  However, BGP is vulnerable to the
   same security attacks that are present in TCP.  The [BGP-VULN]
   explains in depth about the BGP security vulnerability.  At the time
   of this writing, several efforts are underway for creating and
   defining an appropriate security infrastructure within the BGP
   protocol to provide authentication and security for its routing
   information; these efforts include [SBGP] and [SOBGP].
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